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Summary  of  RT  5  SoS  SE  Activities  and 
Future  Recommendations 


Prepared  by  Dr.  Jo  Ann  Lane,  RT  5  SoS  SE  Principal  Investigator 


Introduction 

This  report  summarizes  the  system  of  systems  (SoS)  systems  engineering  (SE)  technical  activities 
performed  under  the  Systems  Engineering  Research  Center  (SERC)  RT  5  task  from  August  2009  through 
October  2010.  In  addition,  recommendations  for  future  SoS  SE  research  are  provided. 


SoS  SE  Activities  Performed 

The  following  sections  identify  the  SoS  SE  activities  performed  within  the  indicated  month. 

August  2009 

SoS  process  research  activities  performed  during  this  period  include:  analysis  of  SoS  processes  with 
respect  to  lean  principles,  analysis  of  modeling  and  simulation  (M&S)  applications  for  SoS  processes 
based  upon  responses  to  a  recent  National  Defense  Industrial  Association  (NDIA)  M&S  survey,  and  SoS 
artifact  analysis  to  support  further  guidance  to  SoS  programs  with  respect  to  content  and  form  of  key 
process  artifacts. 

September  2009 

Continued  analysis  of  SoS  processes  with  respect  to  lean  principles,  analysis  of  M&S  applications  for  SoS 
processes  based  upon  responses  to  a  recent  National  Defense  Industrial  Association  (NDIA)  M&S  survey, 
and  SoS  artifact  analysis  to  support  further  guidance  to  SoS  programs  with  respect  to  content  and  form 
of  key  process  artifacts.  Began  development  of  a  generic  SoS  case  study  to  support  analyses  and 
presentation  of  SoS  research  findings  at  conferences  and  workshops. 

October  2009 

Continued  analysis  of  SoS  processes  with  respect  to  lean  principles,  analysis  of  M&S  applications  for  SoS 
processes  based  upon  responses  to  a  recent  NDIA  M&S  survey,  and  SoS  artifact  analysis  to  support 
further  guidance  to  SoS  programs  with  respect  to  content  and  form  of  key  process  artifacts.  Attended 
NDIA  Systems  Engineering  conference  in  San  Diego,  CA  and  presented  the  results  of  the  M&S  survey  and 
the  use  of  SysML  to  characterize  and  model  SoSs.  Continued  development  of  a  generic  SoS  case  study  to 
support  analyses  and  presentation  of  SoS  research  findings  at  conferences  and  workshops.  (This  model 
is  now  referred  to  as  the  USC  SoS  Reference  Model  and  is  being  used  by  OSD,  USC,  and  MIT  to  further 
the  understanding  of  SoSE.)  Collaborated  with  MIT  Lean  Advancement  Initiative  representatives  to 
develop  an  IEEE  Systems  extended  abstract  on  how  to  accelerate  the  understanding  and  optimization  of 
SoSE  through  lean  enterprise  principles. 
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November  2009 

Continued  analysis  of  SoS  processes  with  respect  to  lean  principles,  analysis  of  SoS  M&S  applications 
based  upon  responses  to  a  recent  NDIA  M&S  survey  and  in  support  of  evolutionary  acquisition  of  cross¬ 
cutting  capabilities,  and  SoS  artifact  analysis  to  support  further  guidance  to  SoS  programs  with  respect 
to  content  and  form  of  key  process  artifacts.  Continued  development  of  a  generic  SoS  case  study  to 
support  analyses  and  presentation  of  SoS  research  findings  at  conferences  and  workshops.  Continued 
collaboration  with  MIT  Lean  Advancement  Initiative  representatives  on  how  to  accelerate  the 
understanding  and  optimization  of  SoSE  through  lean  enterprise  principles. 

Attended  USC  COCOMO  Forum  hosted  by  MIT  the  week  of  November  2nd  and  met  with  several  MIT 
researchers  working  in  the  SoS  arena.  Key  side  meetings  were  with  Prof.  Joseph  Sussman,  Tsoline 
Mikaelian,  and  Chris  Roberts.  Most  of  the  SoS  topics  discussed  in  these  meetings  were  focused  on  SoS 
in  the  transportation  domain,  SoS  organizational  and  management  research,  SoS  cost  modeling  and  SoS 
decision  theory.  In  addition,  met  with  Ray  Madachy  from  NPS  to  discuss  extensions  to  SoS  system 
dynamics  cost  model  to  incorporate  features  to  evaluate  capability  change  rate  and  SoS  quality  factors. 

December  2009 

Continued  analysis  of  SoS  processes  with  respect  to  lean  principles  and  SoS  artifact  analysis  to  support 
further  guidance  to  SoS  programs  with  respect  to  content  and  form  of  key  process  artifacts.  Continued 
development  of  SoS  Reference  Model  to  support  analyses  and  presentation  of  SoS  research  findings  at 
conferences  and  workshops.  Continued  collaboration  with  MIT  Lean  Advancement  Initiative 
representatives  on  how  to  accelerate  the  understanding  and  optimization  of  SoSE  through  lean 
enterprise  principles. 

January  2010 

Continue  analysis  of  SoS  processes  with  respect  to  SoS  artifact  analysis,  SoS  test  and  evaluation, 
development  of  the  SoS  Reference  Model,  and  development  of  extensions  to  SoS  system  dynamics 
model  to  evaluate  the  impact  of  capability  change  rates  and  quality  factors  related  to  the  level  of  SoS 
engineering  activities. 

February  2010 

Continued  analysis  of  SoS  processes  with  respect  to  SoS  artifact  analysis  to  support  further  guidance  to 
SoS  programs  with  respect  to  content  and  form  of  key  process  artifacts.  Continued  development  of  SoS 
and  SoSE  library  to  support  SoSE  research  and  analysis.  Provided  inputs  to  IEEE  SoSE  paper  on  SoS  test 
and  evaluation. 

March  2010 

Continued  analysis  of  SoS  processes  with  respect  to  SoS  artifact  and  SoS  test  and  evaluation.  Finalized 
SoS  test  and  evaluation  paper  for  submission  to  IEEE  SoSE  2010  conference.  Updated  the  MITRE  SoS  and 
SoSE  SharePoint  library  to  support  SoSE  research  and  analysis.  Provided  inputs  to  IEEE  SoSE  paper  on 
SoS  test  and  evaluation.  Developed  a  plan  for  SoSE  artifact  journal  paper  and  survey  form  to  capture 
additional  artifact  information  for  paper.  Conducted  a  literature  review  in  response  to  Mr.  Steve  Welby's 
SoS  SE  questions.  Attended  the  NIDA  Test  and  Evaluation  conference  in  San  Diego,  CA  and  provided  a 
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summary  of  sessions  attended  to  DDR&E  SoS  SE  team.  Attended  the  USC  Center  for  Systems  and 
Software  Engineering  Annual  Research  Review  (ARR),  provided  an  overview  of  on-going  SoSE  research, 
participated  in  the  MIT/USC  PATFrame  workshop  on  unmanned  autonomous  SoS  test  and  evaluation, 
and  provided  summary  of  SoS-related  information  presented  at  the  conference  to  the  DDR&E  SoS  SE 
team. 

April  2010 

Continued  analysis  of  SoS  processes  with  respect  to  SoS  artifact  and  SoS  test  and  evaluation.  Provided 
additional  inputs  to  IEEE  SoSE  paper  and  presentation  on  SoS  test  and  evaluation.  Continued  to  develop 
the  plan  for  SoSE  artifact  journal  paper  and  survey  form  to  capture  additional  artifact  information  for 
paper  and  began  mining  SoS  SE  guidebook  pilot  interview  notes  for  detailed  artifact  data.  Began  review 
of  SoS  SE  guidebook  pilot  interview  notes  for  data  related  to  Mr.  Steve  Welby's  SoS  SE  questions. 
Attended  IEEE  Systems  2010  Conference  in  San  Diego,  CA  and  presented  a  paper  that  analyzed  SoS  SE 
activities  with  respect  to  lean  enterprise  principles.  Provided  a  summary  of  sessions  attended  to  DDR&E 
SoS  SE  team.  Provided  inputs  and  attended  US-Australian  SoS  telecon  on  4/12/2010.  Attended  Army 
ATEC  briefing  at  the  SoS  T&E  telecon  on  4/13/2010.  Attended  the  NDIA  SoS  SE  telecon  on  4/20/2010. 
Presented  "SysML  Strategies  to  Characterize  and  Analyze  Systems  of  Systems"  at  the  SoS  Collaborator's 
meeting  on  4/27/2010. 

May  2010 

Attended  the  International  Conference  on  Software  Engineering  (ICSE)  from  May  4-  7,  2010.  Presented 
Incremental  Commitment  Model  implications  and  support  for  system  of  systems  engineering  and 
contrasted  it  to  other  system  development  special  cases.  Continued  analysis  of  SoS  processes  with 
respect  to  SoS  artifact  and  SoS  test  and  evaluation  and  discussed  findings  at  SoS  SE  telecons  led  by  Dr. 
Judith  Dahmann.  Provided  final  inputs  to  IEEE  SoSE  paper  on  SoS  test  and  evaluation.  Continued 
collection  of  information  for  SoSE  artifact  journal  paper,  primarily  by  mining  SoS  SE  guidebook  pilot 
interview  notes  for  detailed  artifact  data.  Continued  review  of  SoS  SE  guidebook  pilot  interview  notes 
for  data  related  to  Mr.  Steve  Welby's  SoS  SE  questions.  Presented  talk  on  SoSs,  what  they  are  and  how 
to  engineer  them  at  the  San  Diego  INCOSE  Chapter  meeting  on  5/24/2010.  Attended  SoS  Collaborator's 
meeting  on  5/25/2010  that  discusses  SoS  interoperability  challenges  and  approaches.  Prepared  and 
submitted  abstracts  on  SoSE  research  for  consideration  at  the  NDIA  2010  conference. 

June  2010 

Continued  analysis  of  SoS  processes  with  respect  to  SoS  artifact  and  SoS  test  and  evaluation  and 
discussed  findings  at  SoS  SE  telecons  led  by  Dr.  Judith  Dahmann.  Set  up  a  wiki  site  at  USC  to  collect, 
share,  and  collaborate  with  other  SoSE  researchers  on  the  development  of  the  Regional  Area  Crisis 
Management  SoS  (RACRS)  reference  SoS  and  associated  example  engineering  artifacts.  Began  work  on 
RACRS  example  CONOPS  by  evaluating  various  CONOPS  outlines  and  tailoring  one  to  fit  SoS  situation. 
Continued  collection  of  information  for  SoSE  artifact  journal  paper,  primarily  by  mining  SoS  SE 
guidebook  pilot  interview  notes  for  detailed  artifact  data.  Continued  review  of  SoS  SE  guidebook  pilot 
interview  notes  for  data  related  to  Mr.  Steve  Welby's  SoS  SE  questions.  Finalized  plans  to  attend  the 
International  Conference  on  Software  Processes  in  Paderborn,  Germany  and  the  INCOSE  Symposium  in 
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Chicago.  Provided  copies  of  papers  to  be  presented  for  SERC  review  and  approval  (Critical  Success 
Factors  for  Rapid,  Innovative  Solutions;  Evidenced-Based  Software  Processes;  and  Architected  Agile 
Solutions  for  Software-Reliant  Systems).  Had  follow-up  discussions  with  May  SoS  Collaborator  speaker 
on  SoS  interoperability  challenges  and  approaches  to  better  understand  findings  and  scope  of  data 
collection.  Attended  SoS  Collaborator's  presentation  by  Stephan  Blanchette  on  6/8/2010.  Prepared  and 
submitted  abstracts  on  SoSE  research  for  consideration  at  the  NDIA  2010  conference. 

July  2010 

Continued  analysis  of  SoS  processes  with  respect  to  SoS  artifact  and  SoS  test  and  evaluation  and 
discussed  findings  at  SoS  SE  telecons  led  by  Dr.  Judith  Dahmann.  Notified  other  SoS  researchers  about 
the  USC  RACRS  wiki  site  to  collect,  share,  and  collaborate  with  other  SoSE  researchers  on  the 
development  of  the  RACRS  reference  SoS  and  associated  example  engineering  artifacts.  Continued  work 
on  RACRS  example  CONOPS  by  evaluating  various  CONOPS  outlines  and  tailoring  one  to  fit  SoS  situation. 
Continued  collection  of  information  for  SoSE  artifact  journal  paper,  primarily  by  mining  SoS  SE 
guidebook  pilot  interview  notes  for  detailed  artifact  data.  Continued  review  of  SoS  SE  guidebook  pilot 
interview  notes  for  data  related  to  Mr.  Steve  Welby's  SoS  SE  questions.  Attended  the  International 
Conference  on  Software  Processes  in  Paderborn,  Germany  and  the  INCOSE  Symposium  in  Chicago. 
Presented  "Accelerating  System  of  Systems  Engineering  Understanding  and  Optimization  through  Lean 
Enterprise  Principles"  at  the  July  DoD  SoS  Collaborator's  biweekly  telecon.  Prepared  and  submitted  a 
paper  titled  "Using  Models  to  Understand  and  Evolve  SoSs"  for  consideration  at  the  International 
Congress  on  Ultra  Modern  Telecommunications  and  Control  Systems,  Special  Session  on  System  of 
Systems  conference  after  SERC  approval. 

August  2010 

Attended  and  participated  in  the  NDIA  SoS  T&E  Collaborative  Workshop  on  August  17,  2010  at  the 
MITRE  facilities  in  McLean,  VA.  Worked  with  the  MITRE  SoS  SE  team  to  develop  report  on  workshop. 
Worked  with  SoS  SE  team  to  elaborate  SoS  SE  artifacts  for  various  presentations  and  a  journal  article. 
Attended  and  participated  in  the  PATFrame  Workshop  held  August  23-25,  2010  at  MIT  in  Boston,  MA. 

September  2010 

Finalized  SoS  artifacts  presentation  for  the  NDIA  SE  2010  conference  and  submitted  it  to  the  SERC 
review/approval  process.  Participated  in  SoS  SE  teleconferences  to  review  the  SoS  SE  IEEE  Systems  2011 
abstract  on  "Implementation  View  of  Systems  Engineering  for  Systems  of  System"  and  to  refine  briefing 
for  Mr.  Steve  Welby  in  response  to  his  SoS  SE  questions.  Began  update  of  RT  5  Technical  and 
Management  Work  Plan  (A009)  to  reflect  Option  Year  2  activities.  Began  work  on  SoS  Processes  final 
report  (A008). 

October  2010 

Finalized  and  delivered  RT  5  Technical  and  Management  Work  Plan  (A009)  to  reflect  Option  Year  2 
activities.  Reviewed  and  provided  comments  on  two  SoS  SE  guidebook  synopses.  Finalized  and 
delivered  SoS  Processes  final  report  (A008). 
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Attended  NDIA  SE  2010  conference  and  presented  RT  5  research  on  "Key  System  of  Systems  Engineering 
Artifacts  to  Guide  Engineering  Activities".  Also  attended  various  other  SoS-related  presentations  during 
the  week. 


Recommendations  for  Future  SoS  SE  Research 

The  following  recommendations  for  future  SoS  SE  research  are  based  upon  technical  guidance 
requested  at  conferences  and  workshops  by  those  responsible  for  the  management  and  evolution  of 
SoSs  or  identified  as  key  SoS  SE  key  challenges  in  conference  papers  and/or  technical  reports.  Also  note 
that  while  each  of  these  research  areas  can  be  independently  pursued,  there  is  considerable  overlap  in 
several  areas. 

1.  Better  acquisition  approaches  and  processes  for  SoS  capabilities:  Many  SoSs  are  not 
acquisition  programs  (or  programs  of  record),  but  are  "umbrella"  programs  that  incorporate 
multiple  systems  that  are  in  various  stages  of  acquisition  with  the  goal  of  providing  cross-cutting 
SoS  capabilities.  SoS  challenges  related  to  current  acquisition  processes  are  SoS  authority  to 
guide  the  evolution  of  the  SoS  as  well  as  funding  to  support  SoS  initiatives  and  testing  at  the  SoS 
level.  In  addition,  without  a  clear  understanding  of  which  new  programs  are  to  participate  in 
which  SoSs,  the  associated  interoperability  requirements,  and  potential  conflicts,  there  are 
often  missed  opportunities  to  improve  or  provide  additional  SoS  capabilities.  While  it  may  not 
be  desirable  to  always  manage  SoSs  as  programs  of  record,  SoS  SE  research  in  this  area  could 
investigate  alternatives  to  improve  the  current  ad  hoc  process. 

2.  Improved  SoS  test  and  evaluation  methods,  processes,  and  tools:  Much  of  the  testing  at  the 
SoS  level  depends  upon  opportunities  to  influence  single  system  test  events,  to  conduct  tests 
during  various  exercises,  and  to  collect  operational  data  at  various  points  in  time.  This  testing 
strategy  can  limit  the  ability  of  the  SoS  engineers  to  understand  SoS  limits  and  to  probe  for 
undesirable  emergent  behaviors.  SoS  SE  research  in  this  area  can  identify  improved  test  and 
evaluation  methods,  processes,  and  tools  that  strengthen  the  ability  of  the  SoS  engineering 
teams  to  test  SoS  capabilities,  understand  their  performance  with  respect  to  the  associated 
objectives,  identify  and  mitigate  potential  undesirable  emergent  behaviors,  and  instrument 
constituent  systems  to  better  support  SoS  test  efforts. 

3.  Maturation  of  cost  models  to  support  the  SoS  SE:  Initial  parametric  cost  model  research  for 
software  development  and  systems  engineering  has  led  to  several  commercial  cost  models  and 
continues  to  support  the  evolution  of  these  cost  models.  These  cost  models  use  a  set  of 
parameters  that  characterize  the  engineering  product,  the  processes  used  to  develop  the 
product,  and  the  skills  and  experience  levels  of  the  technical  staff  responsible  for  the 
development  of  the  product.  However,  as  systems  are  being  integrated  into  one  or  more  net- 
centric  system  of  systems  (NCSoS)  and  single  system  capabilities  are  being  integrated  to  support 
one  or  more  NCSoS  capabilities,  it  has  become  necessary  to  develop  processes  and  tools  to 
evaluate  the  costs  of  various  approaches  for  providing  these  more  complex,  cross-cutting  SoS 
capabilities.  Initial  work  has  led  to  a  relatively  solid  system  of  systems  engineering  (SoSE)  cost 
model  framework  that  is  based  Department  of  Defense  (DoD)  case  studies  of  SoSE  processes  [1] 
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and  on  previous  software  and  systems  engineering  cost  models  for  single  systems.  The  next 
steps  are  to  collect  data  from  a  selected  set  of  pilot  SoS(s)  to  better  understand  the  predictive 
ability  of  this  SoSE  cost  model  framework  as  well  as  to  further  expand  the  current  cost  model. 
Suggested  research  includes  analysis  and  refinement  of  the  existing  SoS  SE  cost  model,  analysis 
to  better  understand  and  capture  the  impacts  of  interoperability  characteristics  and  constraints 
on  SoSE  costs,  and  development  of  extensions  to  evaluate  SoS  total  cost  of  ownership  and 
incorporate  incremental  productivity  decline  factors  as  an  SoS  becomes  larger  and  more 
complex. 
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